Generating a composite image from a physical item

ABSTRACT

A computer-implemented method includes capturing, with a camera, a first image of a physical item at a first camera position, detecting borders associated with the physical item, based on the first image, generating an overlay that includes a plurality of objects that are positioned within one or more of the borders associated with the physical item, capturing, with the camera, subsequent images of the physical item, where each subsequent image is captured with a respective subsequent camera position, and during capture of the subsequent images, displaying an image preview that includes the overlay. The method further includes establishing correspondence between pixels of the first image and pixels of each of the subsequent images and generating a composite image of the physical item, where each pixel value of the composite image is based on corresponding pixel values of the first image and the subsequent images.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/350,840, filed Nov. 14, 2016 and titled GENERATING ACOMPOSITE IMAGE FROM A PHYSICAL ITEM, which claims priority to U.S.Provisional Patent Application No. 62/347,572, filed Jun. 8, 2016 andtitled GENERATING AN ELECTRONIC IMAGE FROM A PHYSICAL OBJECT, both ofwhich are incorporated herein by reference in their entirety.

BACKGROUND

People frequently own physical photographs that memorialize importantevents in their lives. It may be advantageous to convert the physicalphotographs into electronic versions. For example, the physicalphotographs are vulnerable because people may lose them or they may getdestroyed. People may prefer electronic versions that they could share,search, and organize into electronic photo albums. However, it may bedifficult to convert the physical photographs into attractive images.

When capturing images of the physical photographs with a camera, lightmay reflect off the physical photograph and be captured by the camera insuch a manner that the images includes glare. The glare may be presentdue to the relative position of the light source, physical photograph,and camera. The glare may also be caused by the use of a flash duringimage capture. Glare is interpreted as a low quality image, since imagedetail is missing in portions with glare. For example, FIG. 1illustrates how a converted image 100 may include glare 101 as a resultof overhead light sources on the physical photograph at the time ofcapture.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

SUMMARY

Implementations generally relate to a computer-implemented method togenerate a composite image. The method includes capturing, with acamera, a first image of a physical item at a first camera position. Themethod further includes detecting borders associated with the physicalitem. The method further includes, based on the first image, generatingan overlay that includes a plurality of objects that are positionedwithin one or more of the borders associated with the physical item. Themethod further includes capturing, with the camera, subsequent images ofthe physical item, wherein each subsequent image is captured with arespective subsequent camera position. The method further includesduring capture of the subsequent images, displaying an image previewthat includes the overlay. The plurality of objects remains fixed in theoverlay as the camera is moved. In response to a predetermined number ofcaptures of subsequent images at each respective subsequent cameraposition, an attribute of an object of the plurality of objects isupdated. Updating the attribute of the plurality of objects correspondsto a completion of capture. The method further includes establishingcorrespondence between pixels of the first image and pixels of each ofthe subsequent images. The method further includes generating acomposite image of the physical item, where each pixel value of thecomposite image is based on corresponding pixel values of the firstimage and the subsequent images.

In some implementations, the overlay further comprises a reticle, thereticle moves responsive to moving the camera, and the subsequent imagesare captured responsive to at least a portion of the reticle overlappingwith the object. In some implementations, the overlay further includesan arrow that points to a direction to move the reticle. In someimplementations, responsive to a size of the first image exceeding apredetermined threshold value, dividing the overlay into sections andpositioning each of the plurality of objects in a center of a respectivesection. In some implementations, responsive to at least one of aconfidence value associated with detection of the first image exceedinga threshold confidence value, dividing the overlay into sections andpositioning each of the plurality of objects in a center of a respectivesection. In some implementations, the attribute is color and updatingthe color of the plurality of objects causes an indication of completionto appear. In some implementations, responsive to updating the attributeof the object of the plurality of objects, the object is removed fromthe overlay. In some implementations, each pixel value of the compositeimage is a minimum value of the corresponding pixel values of the firstimage and the subsequent images. In some implementations, generating thecomposite image further includes for each pixel value of the compositeimage, selecting a tentative pixel value and comparing the tentativepixel value to surrounding pixel values for pixels that surround a pixelcorresponding to the pixel value to confirm that each of the surroundingpixel values is within a threshold difference of the tentative pixelvalue. In some implementations, responsive to a difference between thetentative pixel value and one of the surrounding pixel values exceedingthe threshold difference, modifying the tentative pixel value to bewithin the threshold difference.

In some implementations, a system comprises one or more processors and amemory that stores instructions executed by the one or more processors,the instructions comprising: capturing a first image of a physical itemwith a camera, displaying the first image with a color overlay over afirst portion of the first image, capturing one or more subsequentimages of the physical item, wherein each subsequent image is capturedas the camera is moved with reference to the physical item, duringcapture of each of the subsequent images, displaying the first imagewith the color overlay over a second portion of the first image, whereinthe second portion includes the first portion and an additional portiondetermined based on movement of the camera, establishing correspondencebetween pixels of the first image and pixels of each of the one or moresubsequent images, and generating a composite image of the physicalitem, wherein each pixel value of the composite image is based oncorresponding pixel values of the first image and the one or moresubsequent images.

In some implementations, the instructions further comprise convertingthe first image to a preset aspect ratio, responsive to the first imagebeing at least one of upside down and sideways, modifying the firstimage to be right-side up, and applying white balance to correct acomposition of the first image. In some implementations, the overlayfurther includes a target that is positioned to indicate a direction tomove the camera. In some implementations, the target moves as the cameramoves.

In some implementations, the method may include a means for capturing,with a camera, a first image of a physical item at a first cameraposition. The method may further include a means for detecting bordersassociated with the physical item. The method may further include, basedon the first image, a means for generating an overlay that includes aplurality of objects that are positioned within one or more of theborders associated with the physical item. The method may furtherinclude a means for capturing, with the camera, subsequent images of thephysical item, wherein each subsequent image is captured with arespective subsequent camera position. The method may further includeduring capture of the subsequent images, a means for displaying an imagepreview that includes the overlay. The plurality of objects remainsfixed in the overlay as the camera is moved. In response to apredetermined number of captures of subsequent images at each respectivesubsequent camera position, a color of an object of the plurality ofobjects is updated. Updating the color of the plurality of objectscorresponds to a completion of capture. The method may further include ameans for establishing correspondence between pixels of the first imageand pixels of each of the subsequent images. The method may furtherincludes a means for generating a composite image of the physical item,where each pixel value of the composite image is based on correspondingpixel values of the first image and the subsequent images.

The system and methods described below advantageously solve the problemof glare and artifacts that result from capturing images of physicalitems. In addition, the system and methods described below may reduce oreliminate blur from a shaky hand capturing the images.

A technical advantage of the application is that the system and methodsdescribed below generate a user interface that provides guidance to auser to capture images that result in generating a high-qualitycomposite image. For example, the user interface provides indicators foroptimal distance between the physical item and a mobile device thatcaptures the images. Further, the user interface provides indicators toadjust the orientation of the mobile device so that the captured imagedoes not have a trapezoid artifact, for example, by positioning themobile device parallel to the plane of the physical item.

In addition, the system and methods avoid providing complicatedinstructions to a user about how to capture images of a physical item bycreating a game of capturing the images. The user interface guides theuser to move the mobile device horizontally and vertically so thatsubsequent images are captured. For example, the user may enjoy movingthe mobile device to cover dots in an overlay with a reticle. In anotherexample, the user may enjoy moving the mobile device to make it looklike a color is being spray painted onto the image. The user interfacealso guides the user to move the mobile device slowly while images arecaptured to prevent the capture of blurry images.

Another technical advantage is that the system and methods eliminateglare and artifacts by generating a composite image. Multiple images ofthe physical item are captured from different camera positions, forexample, by moving a camera relative to the physical item. The capturedimages are combined and the pixel values are selected from the capturedimages to reduce or eliminate glare. The composite image includes pixelvalues determined from actual captured images and not approximated fromneighboring pixel values that could themselves include glare.

Yet another technical advantage is that image capture can be performedin any lighting condition and does not need specific configuration of alight source, camera, and the physical item. In addition, the system andmethods may be performed by a mobile device instead of specializedequipment, such as a flat-bed scanner.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is illustrated by way of example, and not by way oflimitation in the figures of the accompanying drawings in which likereference numerals are used to refer to similar elements.

FIG. 1 illustrates a prior art graphic representation of a photograph.

FIG. 2 illustrates a block diagram of an example system that generates acomposite image from a physical item according to some implementations.

FIG. 3 illustrates a block diagram of an example computing device thatgenerates a composite image from a physical item according to someimplementations.

FIG. 4A illustrates a graphic representation of an image including fourdots that are each in a detected quadrant.

FIG. 4B illustrates a graphic representation of a mobile devicedisplaying an overlay and an image in a portrait orientation.

FIG. 4C illustrates a graphic representation of a mobile devicedisplaying an overlay and a crooked image.

FIG. 4D illustrates a graphic representation of a mobile devicedisplaying an overlay and a small image.

FIG. 4E illustrates a graphic representation of a mobile devicedisplaying an overlay and no image.

FIG. 4F illustrates a graphic representation of a mobile devicedisplaying an overlay and an image at an angle.

FIG. 4G illustrates a graphic representation of a mobile devicedisplaying an overlay and an image with a landscape orientation.

FIG. 5A illustrates a graphic representation of a mobile devicedisplaying an overlay with four objects and a reticle.

FIG. 5B illustrates a graphic representation of the mobile devicedisplaying the overlay with the reticle surrounding a first object.

FIG. 5C illustrates a graphic representation of the mobile devicedisplaying the overlay with the reticle surrounding a second object.

FIG. 5D illustrates a graphic representation of the mobile devicedisplaying the overlay with the reticle surrounding a third object.

FIG. 5E illustrates a graphic representation of the mobile devicedisplaying the overlay with the reticle surrounding a fourth object.

FIG. 5F illustrates a graphic representation of the mobile devicedisplaying a completion of the capture process.

FIG. 6A illustrates a graphic representation of a mobile devicedisplaying a color overlay over a first portion of an image according tosome implementations.

FIG. 6B illustrates a graphic representation of an example userinterface with the color overlay over a second portion of the imageaccording to some implementations.

FIG. 6C illustrates a graphic representation of an example userinterface with the color overlay over all of the image according to someimplementations.

FIG. 7 illustrates a flowchart of an example method to generate acomposite image from a physical item according to some implementations.

FIG. 8 illustrates a flowchart of another example method to generate acomposite image from a physical item according to some implementations.

DETAILED DESCRIPTION

In some implementations, a camera captures a first image of a physicalitem. For example, the physical item is a photograph. Borders associatedwith the physical item are detected. For example, photographs aretypically shaped like a rectangle and the borders may be of a rectangle.An overlay may be generated based on the first image that includesobjects and a reticle where the objects are positioned within theborders associated with the physical item. For example, the first imageincludes four blue dots positioned at the edges of a rectangle and areticle in the center.

Subsequent images of the physical item are captured, where eachsubsequent image is captured with a respective subsequent cameraposition. For example, a user may tilt and rotate the camera to capturesubsequent images at different angles. During capture of the subsequentimages, an image preview that includes the overlay is displayed. Theplurality of objects remain fixed in the overlay as the camera is moved.The reticle moves responsive to moving the camera. The subsequent imagesare captured responsive to at least a portion of the reticle overlappingwith one of the objects. For example, the user may move the camera to anobject in the lower left-hand corner.

In response to a predetermined number of captures of subsequent imagesat each respective subsequent camera position, an attribute of theobject is updated. For example, the blue dot changes to a gray dot. Acompletion of capture occurs responsive to moving the reticle until theattribute of the plurality of objects is updated. For example, the usermoves the reticle from a first object to a second object in the lowerright-hand corner, then to a third object in the upper right-handcorner, and then to a fourth object in the upper left-hand corner. Onceall the objects change to gray, the capture is complete. In someimplementations, after the color of the objects is updated, anindication of completion appears. For example, a greet circle with acheckmark is displayed.

Example System

FIG. 2 illustrates a block diagram of an example system 200 thatgenerates composite images. The illustrated system 200 includes a mediaserver 111, mobile devices 115 a, 115 n, and a network 105. Users 125 a,125 n may be associated with respective mobile devices 115 a, 115 n. Insome implementations, the system 200 may include other servers ordevices not shown in FIG. 1. In FIG. 1 and the remaining figures, aletter after a reference number, e.g., “115 a,” represents a referenceto the element having that particular reference number. A referencenumber in the text without a following letter, e.g., “115,” represents ageneral reference to implementations of the element bearing thatreference number.

In the illustrated implementation, the entities of the system 200 arecommunicatively coupled via a network 105. The network 105 may be aconventional type, wired or wireless, and may have numerous differentconfigurations including a star configuration, token ring configurationor other configurations. Furthermore, the network 105 may include alocal area network (LAN), a wide area network (WAN) (e.g., theInternet), and/or other interconnected data paths across which multipledevices may communicate. In some implementations, the network 105 may bea peer-to-peer network. The network 105 may also be coupled to orinclude portions of a telecommunications network for sending data in avariety of different communication protocols. In some implementations,the network 105 includes Bluetooth® communication networks, WiFi®, or acellular communications network for sending and receiving data includingvia short messaging service (SMS), multimedia messaging service (MMS),hypertext transfer protocol (HTTP), direct data connection, email, etc.Although FIG. 1 illustrates one network 105 coupled to the mobiledevices 115 and the media server 111, in practice one or more networks105 may be coupled to these entities.

The media server 111 may include a processor, a memory, and networkcommunication capabilities. In some implementations, the media server111 is a hardware server. The media server 111 is communicativelycoupled to the network 105 via signal line 102. Signal line 102 may be awired connection, such as Ethernet, coaxial cable, fiber-optic cable,etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or otherwireless technology. In some implementations, the media server 111 sendsand receives data to and from one or more of the mobile devices 115 a,115 n via the network 105. The media server 111 may include a mediaapplication 103 a and a database 199.

The media application 103 a may be code and routines operable togenerate composite images. In some implementations, the mediaapplication 103 a may be implemented using hardware including afield-programmable gate array (FPGA) or an application-specificintegrated circuit (ASIC). In some implementations, the mediaapplication 103 a may be implemented using a combination of hardware andsoftware.

The database 199 may store images captured by users 125 associated withmobile devices 115 and composite images generated from the images. Insome implementations, the database 199 may store composite images thatwere generated independent of the mobile devices 115. The database 199may also store social network data associated with users 125, userpreferences for the users 125, etc.

The mobile device 115 may be a computing device that includes a memory,a hardware processor, and a camera. For example, the mobile device mayinclude a tablet computer, a mobile telephone, a wearable device, ahead-mounted display, a mobile email device, a portable game player, aportable music player, a reader device, or another electronic devicecapable of accessing a network 105.

In the illustrated implementation, mobile device 115 a is coupled to thenetwork 105 via signal line 108 and mobile device 115 n is coupled tothe network 105 via signal line 110. Signal lines 108 and 110 may be awired connection, such as Ethernet, coaxial cable, fiber-optic cable,etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or otherwireless technology. Mobile devices 115 a, 115 n are accessed by users125 a, 125 n, respectively. The mobile devices 115 a, 115 n in FIG. 1are used by way of example. While FIG. 1 illustrates two mobile devices,115 a and 115 n, the disclosure applies to a system architecture havingone or more mobile devices 115.

In some implementations, the mobile device 115 can be a mobile devicethat is included in a wearable device worn by the user 125. For example,the mobile device 115 is included as part of a clip (e.g., a wristband),part of jewelry, or part of a pair of glasses. In another example, themobile device 115 can be a smart watch. The user 125 may view imagesfrom the media application 103 on a display of the device worn by theuser 125. For example, the user 125 may view the images on a display ofa smart watch or a smart wristband.

In some implementations, media application 103 b may be stored on amobile device 115 a. The media application 103 may include a thin-clientmedia application 103 b stored on the mobile device 115 a and a mediaapplication 103 a that is stored on the media server 111. For example,the media application 103 b stored on the mobile device 115 a maycapture the images that are transmitted to the media application 103 astored on the media server 111 where a composite image is generated fromthe images. The media application 103 a may transmit the composite imageto the media application 103 b for display on the mobile device 115 a.In another example, the media application 103 b stored on the mobiledevice 115 a may generate the composite image and transmit the compositeimage to the media application 103 a stored on the media server 111. Themedia application 103 a stored on the media server 111 may include thesame components or different components as the media application 103 bstored on the mobile device 115 a.

In situations in which the systems and methods discussed herein maycollect or use personal information about users (e.g., user data,information about a user's social network, user's location, user'sbiometric information, user's activities and demographic information,the media server stores and analyzes videos), users are provided withopportunities to control whether information is collected, whether thepersonal information is stored, whether the personal information isused, whether the videos are analyzed, and how the information about theuser is collected, stored, and used. That is, the systems and methodsdiscussed herein collect, store, and/or use user personal informationonly upon receiving explicit authorization from the relevant users to doso. For example, a user is provided with control over whether programsor features collect user information about that particular user or otherusers relevant to the program or feature. Each user for which personalinformation is to be collected is presented with one or more options toallow control over the information collection relevant to that user, toprovide permission or authorization as to whether the information iscollected and as to which portions of the information are to becollected. For example, users can be provided with one or more suchcontrol options over a communication network. In addition, certain datamay be treated in one or more ways before it is stored or used so thatpersonally identifiable information is removed. As one example, a user'sidentity information may be treated, e.g., anonymized, so that nopersonally identifiable information can be determined from a video. Asanother example, a user's geographic location may be generalized to alarger region so that the user's particular location cannot bedetermined.

Example Computing Device

FIG. 3 illustrates a block diagram of an example computing device 300that generates composite images. The computing device 300 may be a mediaserver 111 or a mobile device 115. The computing device 300 may includea processor 235, a memory 237, a communication unit 239, a display 241,a camera 243, a sensor 245, and a storage device 247. Additionalcomponents may be present or some of the previous components may beomitted depending on the type of computing device 300. For example, ifthe computing device 300 is the media server 111, the computing device300 may not include the display 241, the camera 243, or the sensor 245.A media application 103 may be stored in the memory 237. Inimplementations where the computing device 300 is a wearable device, thecomputing device 300 may not include storage device 247. In someimplementations, the computing device 300 may include other componentsnot listed here, such as a battery, etc. The components of the computingdevice 300 may be communicatively coupled by a bus 220.

The processor 235 includes an arithmetic logic unit, a microprocessor, ageneral purpose controller or some other processor array to performcomputations and provide instructions to a display device. Processor 235processes data and may include various computing architectures includinga complex instruction set computer (CISC) architecture, a reducedinstruction set computer (RISC) architecture, or an architectureimplementing a combination of instruction sets. Although FIG. 2 includesa single processor 235, multiple processors 235 may be included. Otherprocessors, operating systems, sensors, displays and physicalconfigurations may be part of the computing device 300. The processor235 is coupled to the bus 220 for communication with the othercomponents via signal line 222.

The memory 237 stores instructions that may be executed by the processor235 and/or data. The instructions may include code for performing thetechniques described herein. The memory 237 may be a dynamic randomaccess memory (DRAM) device, a static RAM, or some other memory device.In some implementations, the memory 237 also includes a non-volatilememory, such as a (SRAM) device or flash memory, or similar permanentstorage device and media including a hard disk drive, a compact discread only memory (CD-ROM) device, a DVD-ROM device, a DVD-RAM device, aDVD-RW device, a flash memory device, or some other mass storage devicefor storing information on a more permanent basis. The memory 237includes code and routines operable to execute the media application103, which is described in greater detail below. The memory 237 iscoupled to the bus 220 for communication with the other components viasignal line 224.

The communication unit 239 transmits and receives data to and from atleast one of the mobile device 115 and the media server 111 dependingupon where the media application 103 may be stored. In someimplementations, the communication unit 239 includes a port for directphysical connection to the network 105 or to another communicationchannel. For example, the communication unit 239 includes a universalserial bus (USB), secure digital (SD), category 5 cable (CAT-5) orsimilar port for wired communication with the mobile device 115 or themedia server 111, depending on where the media application 103 may bestored. In some implementations, the communication unit 239 includes awireless transceiver for exchanging data with the mobile device 115,media server 111, or other communication channels using one or morewireless communication methods, including IEEE 802.11, IEEE 802.16,Bluetooth® or another suitable wireless communication method. Thecommunication unit 239 is coupled to the bus 220 for communication withthe other components via signal line 226.

In some implementations, the communication unit 239 includes a cellularcommunications transceiver for sending and receiving data over acellular communications network including via short messaging service(SMS), multimedia messaging service (MMS), hypertext transfer protocol(HTTP), direct data connection, e-mail or another suitable type ofelectronic communication. In some implementations, the communicationunit 239 includes a wired port and a wireless transceiver. Thecommunication unit 239 also provides other conventional connections tothe network 105 for distribution of files and/or media objects usingstandard network protocols including, but not limited to, user datagramprotocol (UDP), TCP/IP, HTTP, HTTP secure (HTTPS), simple mail transferprotocol (SMTP), SPDY, quick UDP internet connections (QUIC), etc.

The display 241 may include hardware operable to display graphical datareceived from the media application 103. For example, the display 241may render graphics to display an overlay and a resulting compositeimage. The display 241 is coupled to the bus 220 for communication withthe other components via signal line 228.

The camera 243 may include hardware operable to capture images ofphysical items. For example, the camera 243 may include a lens, asensor, and an image signal processor. The lens may capture an opticalimage. The sensor may convert the optical image to electrical signals.The image signal processor may perform demosaicing on the electricalsignals to determine pixel color and control image effects, such asautofocus, exposure, and white balance. The camera 243 may transmit theimages to the media application 103 or to the storage device 247. Thecamera 243 may be coupled to the bus 220 via signal line 230.

The sensor 245 may include hardware operable to determine a change inposition of the mobile device 115. For example, the sensor 245 mayinclude motion sensors that measure acceleration forces and rotationalforces along the x, y, and z-axes, such as an accelerometer and agyroscope. The sensor 245 may also include position sensors that measurethe physical position of the mobile device 115, such as orientationsensors and magnetometers. The sensor 245 is coupled to the bus 220 forcommunication and with the other components via signal line 232.

The storage device 247 may be a non-transitory computer-readable storagemedium that stores data that provides the functionality describedherein. In implementations where the computing device 300 is the mediaserver 111, the storage device 247 may include the database 199 inFIG. 1. The storage device 247 may be a DRAM device, a SRAM device,flash memory or some other memory device. In some implementations, thestorage device 247 also includes a non-volatile memory or similarpermanent storage device and media including a hard disk drive, a CD-ROMdevice, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flashmemory device, or some other mass storage device for storing informationon a permanent basis. The storage device 247 is coupled to the bus 220for communication with the other components via signal line 234.

The media application 103 may include a capture module 202, a userinterface module 204, and a composite module 206.

The capture module 202 processes an image captured by the camera 243. Insome implementations, the capture module 202 includes a set ofinstructions executable by the processor 235 to process the images. Insome implementations, the image processing module 202 is stored in thememory 237 of the computing device 300 and can be accessible andexecutable by the processor 235.

The capture module 202 receives a first image of a physical item at afirst camera position from the camera 243. For example, a user of amobile device 115 that includes a camera 243 positions the mobile device115 over a physical item and selects an option to capture the firstimage. The physical item may include, for example, a photograph, abillboard, a document, a restaurant menu, etc.

The capture module 202 may perform processing of the first image. Forexample, the capture module 202 may crop the first image or modify thefirst image to fit a particular aspect ratio. The aspect ratio mayinclude, for example, 1:1, 4:3, 16:9, etc. The capture module 202 mayidentify a direction of the first image (e.g., upside down, sideways,etc.) and rotate the first image to be right-side up. The capture module202 may apply white balance to correct a composition of the first image.

The capture module 202 detects borders associated with the physicalitem. For example, the capture module 202 may determine that thephysical item has a rectangular shape with four borders. Other shapesare possible. For example, the physical item may have a triangularshape, a square shape, a circular shape, an oval shape, a rhombus shape,etc. The user interface module 204 may include user instructions on howto position the mobile device 115 over the physical item. For example,the instructions may include the following text: “Position your photosuch that all four corners can be seen.” This may help the capturemodule 202 to detect the borders.

The capture module 202 instructs the user interface module 204 todisplay an overlay with objects that are positioned within the bordersassociated with the physical item. In some implementations, the capturemodule 202 divides the first image into sections and instructs the userinterface module 204 to display an overlay with objects that are each inthe center of one of the sections. For example, photographs may bedivided into quadrants and each quadrant may include a dot at the centerof each of the quadrants. In another example, the first image may bedivided into a different number of sections, such as five sections, or atriangular-shaped physical item may be divided into three differentlyshaped sections, etc. FIG. 4A illustrates a graphic representation 400of an image including four dots 405 a, 405 b, 405 c, 405 d that are eachin a detected quadrant 410 a, 410 b, 410 c, 410 d. The objects mayinclude a variety of different shapes, such as circles, ovals, squares,rectangles, parallelograms, etc.

The capture module 202 may determine a size of the first image. In someimplementations, the capture module 202 may determine the size of thefirst image as a percentage of the area of the screen of the mobiledevice 115. If the size of the first image exceeds a predeterminedthreshold value, the capture module 202 may instruct the user interfacemodule 204 to display an overlay that is divided into sections with anobject at the center of the section. If the size of the first imagefails to exceed the predetermined threshold value, the capture module202 may instruct the user interface module 204 to include an overlaywith default objects, such as dots. The default objects may not alignwith divided sections of the first image, but may help guide a user toposition the mobile device 115 in a way to capture subsequent images forcreating a composite image. In some implementations, if the defaultobjects result in a composite image that includes extraneous content,such as a background behind a physical item, the user interface module204 may include a cropping option for cropping out the extraneouscontent.

The capture module 202 may assign a confidence value associated withdetection of the first image. The confidence value may reflect aconfidence of a detection of the borders of the physical item. If theconfidence value exceeds a predetermined threshold value, the capturemodule 202 may instruct the user interface module 204 to display anoverlay that is divided into sections with an object at the center ofthe section. If the confidence value fails to exceed the thresholdconfidence value, the capture module 202 may instruct the user interfacemodule 204 to include an overlay with default objects. In someimplementations, the overlay may be divided into sections based on acombination of the size of the first image and a confidence valueassociated with the first image.

In some implementations, the overlay includes a reticle. The reticle mayform a variety of shapes, however, the reticle is displayed as anoutline. The reticle may stay a fixed size, regardless of the size ofthe objects in the overlay. In some implementations, the reticle movesresponsive to a user moving a mobile device 115. Moving the reticle sothat some or all of the reticle overlaps with one of the objects causesthe mobile device 115 to capture a subsequent image. For example, wherethe first image is divided into four quadrants, moving the reticle tooverlap with the four dots results in four subsequent images beingcaptured by the mobile device 115. The objects may remain in fixedpositions in the overlay as the mobile device 115 is moved (e.g., withrespect to a displayed image of the physical item). The size of theobject may change depending on a distance between the mobile device 115and the physical item. For example, the size of the object may increaseas the mobile device 115 moves closer to the physical item to act as aguide for how close the mobile device 115 gets to the physical item.Guiding the user to move the mobile device 115 closer to physical itemsadvantageously ensures that the images are of a higher quality than ifthe user kept the mobile device 115 further away from the physical item.

FIG. 4B illustrates a graphic representation 425 of a mobile devicedisplaying an overlay 426 and an image. In this example, a reticle 427is included in the center of the overlay 426 as a starting point. Oncethe first image is captured, the user interface module 204 displays animage preview that includes the overlay. Moving the mobile device 428causes the reticle 427 to move. The objects 429 a, 429 b, 429 c, and 429d remain stationary while the reticle 427 moves. As a result, thereticle 427 may be moved to surround or overlap the objects 429 a, 429b, 429 c, and 429 d.

In some embodiments, once the reticle 427 surrounds or overlaps with anobject 429, the reticle 427 is illustrated as a spinner thatprogressively changes color to illustrate an amount of time needed forthe mobile device 115 to capture a predetermined number of captures ofsubsequent images at each respective subsequent camera position. Forexample, the camera 243 may capture a high-quality subsequent image foreach quadrant while the mobile device 115 is positioned at eachrespective subsequent camera position. In some implementations, thecamera 243 may capture the subsequent images until an event other than apredetermined number of subsequent images are captured. For example, thecamera 243 may capture the subsequent images until the capture module202 determines that a subsequent image is associated with a qualityscore that exceeds a predetermined value, a predetermined amount of timehas passed, etc.

While the camera 243 captures the predetermined number of subsequentimages, the reticle 427 may switch from a completely white outline to aportion of the white outline changing to blue (or another color) andprogressively becoming more blue in a clockwise direction until apredetermined amount of time has passed, a predetermined number ofsubsequent images are captured, or the captured module 202 receives asignal from the camera 243 indicating that the subsequent image has beencaptured. In some implementations, an attribute of the object 429 thatthe reticle 427 surrounds or overlaps changes once the subsequent imageis captured. For example, the object may change color, the object mayflash, or the object may change size. When the attribute of the objectchanges, it may signal a completion of capture for the correspondingsection. For example, in FIG. 4B, the reticle 427 surrounds the firstobject 429 a until the subsequent image for the first quadrant iscaptured and the color of the object 429 a changes from gray to blue. Insome implementations where the user moves the camera 243 away from thephysical item before the camera 243 has finished capturing thesubsequent image or subsequent images, the capture module 202 mayinstruct the user interface module 204 to keep the reticle white as anindication that the capturing needs to be repeated for the correspondingobject. In some implementations, if the user does not move the camera243 to surround one of the objects with the reticle for more than apredetermined amount of time, such as 10 seconds, the capturing may becancelled and the capture module 202 instructs the user interface module204 to display instructions to the user to start the capturing overagain. For example, the instructions may include: “Scan cancelled. Movecircle to dot to finish scanning next time.”

In some implementations, the objects in the overlay are parallel to thedetected borders. Thus, if the physical item is captured at an angle,the objects in the overlay may be displayed at an angle as well. Forexample, FIG. 4C illustrates a graphic representation 430 of a mobiledevice 433 displaying an overlay and a crooked image. Because thereticle 431 is a fixed size while the objects 432 change size based onthe size of the image, the size difference between the reticle 431 andthe objects 432 provide a visual indicator to a user that the image istoo small and that the mobile device 433 is to be moved closer to thephotograph.

In some implementations, if the capture module 202 determines that thesize of the image fails to exceed a predetermined threshold value, thecapture module 202 may instruct the user interface module 204 to displaydefault objects and/or to omit the reticle from the overlay. Forexample, FIG. 4D illustrates a graphic representation 440 of a mobiledevice 441 displaying an overlay and a small image 442. Because the sizeof the image fails to exceed the predetermined threshold, the objects443 are default objects. The default objects are not at the center ofeach quadrant, however, the mobile device 115 may still capture thefirst image and the subsequent images. In another example, FIG. 4Eillustrates a graphic representation 450 of a mobile device 451displaying an overlay and no image. In this example, the capture module202 instructs the user interface module 204 to display the overlay withthe objects 452 even though no image is detected.

In some implementations, objects in the overlay may change shape as themobile device 115 is moved. The shape may serve as an indicator to auser of the mobile device 115 to move the mobile device 115 to obtainoptimal images. In some implementations, if the mobile device 115 istiled more than 20 degrees relative to the physical item, the capturemodule 202 instructs the user interface module 204 to display defaultobjects.

FIG. 4F illustrates a graphic representation 460 of a mobile device 461displaying an overlay and an image at an angle. In this example, themobile device 461 is positioned at an angle relative to the photograph.The objects 462 a, 462 b at the top of the overlay form oval shapes toindicate that the mobile device 461 is to be tilted so that the mobiledevice 461 is parallel to the photograph. The objects 463 a, 463 b atthe bottom of the overlay form ovals that are closer to circles toindicate that the bottom position of the mobile device 115 should onlybe tilted slightly to be parallel with the photograph.

The capture module 202 detects the borders of physical items withdifferent shapes and orientations. For example, FIG. 4G illustrates agraphic representation 470 of a mobile device 471 displaying an overlayand an image with a landscape orientation. The capture module 202identifies quadrants for the rectangular shape of the image andgenerates four objects 472 that are each positioned in the center ofeach quadrant.

In some implementations, the capture module 202 instructs the userinterface module 204 to generate an overlay that is first displayed overa portion of an image and that increasingly covers the image as themobile device 115 is moved and the camera 243 captures subsequentimages. For example, the overlay may be a color overlay and the processmay be presented as a game where a user moves the mobile device 115 asif spray painting a physical item. These implementations are discussedin greater detail below with reference to FIGS. 6A-6C.

The user interface module 204 generates a user interface. In someimplementations, the user interface module 204 includes a set ofinstructions executable by the processor 235 to generate the userinterface. In some implementations, the user interface module 204 isstored in the memory 237 of the computing device 300 and can beaccessible and executable by the processor 235.

In some implementations, the user interface module 204 receivesinstructions from the capture module 202 to generate a user interfacethat includes an overlay that includes objects that are positionedwithin one or more of the borders associated with an image of a physicalitem. As the camera 243 captures subsequent images of the physical item,where each subsequent image is captured with a respective subsequentcamera position, the user interface module 204 displays an image previewthat includes the overlay. The plurality of objects remain fixed in theoverlay as the camera is moved. In response to a predetermined number ofcaptures of subsequent images at each respective subsequent cameraposition, the user interface module 204 may update an attribute of anobject, such as changing the color of the object, causing the object toflash, changing the size of the object, etc. The user interface module204 may continue to update the attribute of subsequent objects until allthe objects are updated, which corresponds to a completion of capture.In some implementations, the user interface module 204 also generates areticle that is part of the overlay and displays the reticle as movingresponsive to the mobile device 115 moving. In this example, the colorof an object is updated after the reticle surrounds or overlaps with theobject and the predetermined number of captures of subsequent images ateach respective subsequent camera position occurs. The process ofcapturing the first image and subsequent images until completion isillustrated in FIGS. 5A-5F as discussed in greater detail below.

FIG. 5A illustrates a graphic representation 500 of a mobile device 501displaying an overlay with four objects 502 a, 502 b, 502 c, 502 d and areticle 503 over a first image. In this example, the capture module 202detected four borders and divided the first image into quadrants. Thecapture module 202 instructed the user interface module 204 to generatea user interface that includes an overlay that includes four objects 502a, 502 b, 502 c, 502 d that are each at or near the center of one of thequadrants. The capture module 202 instructed the user interface module204 to include a reticle 503. The reticle 503 is larger than the objects502 because the mobile device 501 is parallel to a photograph and withina distance from the photograph that results in capturing a first imageand subsequent images that are high-quality images. The overlay alsoincludes an arrow 504 that points to a direction to move the reticle. Inthis example, the arrow 504 points to the first object 502 a in thelower left-hand corner of the overlay. The user interface also includesa stop icon 505 that, responsive to selection by a user, stops theprocess. Selection of the stop icon 505, e.g., by a user touching thescreen at the location of the stop icon 505, may pause the process orstop the process entirely. The stop icon 505 stays in a fixed positionwithin the user interface.

FIG. 5B illustrates a graphic representation 510 of the mobile device501 displaying the overlay with the reticle 503 surrounding oroverlapping with a first object 502 a. In this example, once the reticle503 surrounds the first object 502 a, the size of the first object 502 ais enlarged to provide an indication to a user moving the mobile device501 that the reticle 503 is recognized as surrounding the first object502. In addition, the user interface module 204 update the first object502 a and the reticle 503 to a different color, such as blue to indicatethat the subsequent image or subsequent images for the lower left-handquadrant have been captured.

FIG. 5C illustrates a graphic representation 520 of the mobile device501 displaying the overlay with the reticle 503 surrounding a secondobject 502 b. In this example, the first object is removed from theoverlay because the subsequent image or subsequent images for thecorresponding quadrant have been captured. The arrow 504 points to thenext object for the reticle 503 to surround, which is the third object502 c. FIG. 5D illustrates a graphic representation 530 of the mobiledevice 501 displaying the overlay with the reticle surrounding a thirdobject 502 c. FIG. 5E illustrates a graphic representation 540 of themobile device 501 displaying the overlay with the reticle surrounding afourth object 502 d.

FIG. 5F illustrates a graphic representation 550 of the mobile device501 displaying a completion of the capture process. In this example, theuser interface module 204 replaces the fourth object with an indicationof completion in the form of a checkmark icon 551. Other indications ofcompletion are possible, such as a thumbs-up icon, a green color, asmiley face, etc. The user interface module 204 replaces the stop iconwith another indication of completion in the form of a circle icon 552,such as a green circle. In this example, the composite module 206generated a composite image and the user interface module 204 displays acomposite image icon 553 that indicates that the composite image wasgenerated and is available (e.g., stored) in mobile device 501. Forexample, the composite image may be stored in a camera roll, an imagegallery, within a camera application, etc.

In some implementations, the user interface module 204 displays thefirst image with a color overlay over a first portion of the firstimage. For example, the user interface module 204 generates a userinterface that includes the first image and the color overlay at thebottom (or the top, middle, etc.) of the first image. The color may beany color, such as blue, black, green, etc. The process of capturing thefirst image and subsequent images until completion is illustrated inFIGS. 6A-6C as discussed in greater detail below.

Turning to FIG. 6A, a graphic representation 600 of a mobile device 601displaying a color overlay over a first portion of an image isillustrated. In this example, the color overlay includes about 40% ofthe image. The top of the color overlay is delineated with arc 602. Inthis example, the color overlay includes a target 604 that may move asthe mobile device 601 moves. For example, if the user moves the mobiledevice 601 up, the target 604 may move upwards. In some implementations,the target 604 is positioned to indicate to the user the direction tomove the mobile device 601. For example, in this example, the target 604may be positioned to indicate to the user that the mobile device 601should be moved upwards.

One or more subsequent images of the physical item may be captured,where each subsequent image is captured as the camera 243 is moved withreference to the physical item. For example, the user may move themobile device 115 from left to right, up and down, or any otherdirection. In some implementations, the user moves the mobile device 115as if the user was spray painting over the physical item. In someimplementations, the user moves the mobile device 115 to capture amoving target. For example, the color overlay may include an icon of anobject, such as a bee, and the user moves the mobile device 115 tocapture the bee. The user interface module 204 may modify the locationof the bee to ensure that the mobile device 115 is moved in a way tocapture the subsequent images that cover different rotations and tiltsto avoid glare or light reflection.

During capture of each of the subsequent images, the first image may bedisplayed with the color overlay over a second portion of the firstimage. The second portion may include the first portion and anadditional portion determined based on movement of the camera. Turningto FIG. 6B, a graphic representation 650 of the color overlay over asecond portion of the image is illustrated. In this example, the coloroverlay covers about 90% of the image and is delineated with the arc652. The user may have used a left to right motion of the mobile device601 to color the first image.

Turning to FIG. 6C, a graphic representation 675 of an example userinterface with the color overlay over all of the image is illustrated.In this example, where the color overlay covers about 100% of the firstimage, the color overlay may indicate to the user that the spraypainting process is complete.

In some implementations, the user interface module 204 generates a userinterface that includes options for generating an electronic photo albumfrom the composite image. For example, a user may use the process forgenerating a composite image for a box of old photos. The user interfacemodule 204 may generate options for combining the composite images intodifferent photo albums, adding titles to each of the composite images,upon user consent, tagging other users in the images, etc. The userinterface module 204 may also generate options for editing the compositeimages, such as cropping, modifying a look of the composite images(e.g., changing color saturation, exposure, etc.), applying a filter,etc.

The composite module 206 generates a composite image. In someimplementations, the composite module 206 includes a set of instructionsexecutable by the processor 235 to generate the composite image. In someimplementations, the composite module 206 is stored in the memory 237 ofthe computing device 300 and can be accessible and executable by theprocessor 235.

The composite module 206 establishes correspondence between pixels ofthe first image and pixels of each of the subsequent images. Thecomposite module 206 may use homography to determine how the first imageand the subsequent images are related. For example, the composite module206 may generate a homography matrix based on rotation and translationbetween the images. The composite module 206 may use optical flow todetermine apparent motion caused by relative motion between the camera243 and the physical item. For example, the media application 103 mayuse optical flow to determine the motion that occurs between two imageswhile the images were captured by the camera 243 during the capture ofthe first image and the subsequent images. The composite module 206 mayuse optical flow to identify glare on the image as a moving part.

The composite module 206 may use image registration to transform thepixels from the first image and the subsequent images into a singlecoordinate system. For example, the composite module 206 may generate acoordinate system that maps the location of each of the pixels based onthe first image and the subsequent images. The composite module 206 maycompare the pixel values for each pixel location and select a pixelvalue for the pixel location based on different factors. For example,the composite module 206 may determine that some of the pixel valuesrepresent glare. As a result, the composite module 206 may select thepixel value that corresponds to the darkest pixel since the other pixelvalue or values may represent glare instead of the pixel value for thecolor beneath the glare. In another example, the composite module 206may determine that none of the pixel values for a particular pixellocation represent glare. As a result, the composite module 206 mayselect the pixel value that represents the lightest pixel so that moredetail from the physical item is represented in a composite image. Insome implementations, the composite module 206 may select the pixelvalues to minimize blurriness that results from a shaky hand capturingthe image.

The composite module 206 may generate a composite image of the physicalitem where each pixel value of the composite image is selected based oncorresponding pixel values of the first image and the subsequent images.In some implementations, the composite module 206 uses sensor data toalign the first image and the subsequent images to each other toidentify the locations of the pixels. For example, the composite module206 receives sensor data from the sensor 245 in the mobile device 115,such as sensor data from a gyroscope, accelerometer, etc. to determine aposition of the mobile device 115 during capture and uses the sensordata to align the images. In some implementations, the images arealigned based on segmentation of the images. In some implementations,the composite module 206 uses the detected borders to align the firstimage and the subsequent images.

In some implementations, the composite module 206 avoids glare byselecting the minimum value of the corresponding pixel values. Otheroptions are possible, such as performing statistical measures thatinclude averaging the pixel values or obtaining a median value of thepixel values.

In some implementations, the composite module 206 may further selecteach pixel value by selecting a tentative pixel value and comparing thetentative pixel value to each of the surrounding pixel values to confirmthat the pixel values are within a threshold difference of each other.If a tentative pixel value is different from one or more of thesurrounding pixel values by more than a threshold difference, thecomposite module 206 may select a different pixel value that is withinthe threshold difference. This may ensure that the pixel values do notcontain outlier pixel values.

The composite module 206 may combine the pixel values for each locationin the first image and the subsequent images to form the compositeimage. The composite module 206 may store the composite image as a datafile in an image format, such as PNG, PG, GIF, etc. In someimplementations, the composite module 206 discards the first image andthe subsequent images after the composite image is generated. In someimplementations, the composite module 206 stores the first image and thesubsequent images, either on the mobile device 115 or on the mediaserver 101.

Example Concept Movies

In some implementations, the media application 103 generates conceptmovies from media associated with a user. The media application 103 mayidentify images and/or videos associated with a user that are associatedwith a particular concept. For example, the media application 103 may,upon user consent, identify images and/or videos of a single childspread over a length of time, such as five years. The concept may be“They Grow Up So Fast” and the images and videos may be organizedchronologically to show the child aging. The media application 103 maygenerate a movie from the images and/or videos and add music to themovie. For example, the media application 103 may generate a movie thattransitions between the images and/or the videos and is synchronizedwith the music, such as based on beats of the music. The mediaapplication 103 may select music that is emotional and corresponds tofeelings evoked by watching the child aging, such as an emotional songdesigned to pull at the heartstrings.

In another example, the concept may be “Summer of Smiles” and on thelast day of summer, the media application 103 may, upon user consent,select images and/or videos from the last three months where people aresmiling. The music selected for the concept may be a song traditionallyassociated with summer.

Example Media Sharer

In some implementations, the media application 103 may make sharingmedia easier for users. The media application 103 may generate a userinterface that provides a user with the user's media and an option toselect multiple media items to share with other users. In someimplementations, the media application 103 may provide an indicator oneach of the media items that the user may select to indicate a desire toshare. For example, the indicator may be a circle or a box and the mediaapplication 103 may add a checkmark responsive to the user selecting themedia item by touching the screen.

In some implementations, the media application 103 may provide the userwith an option to share the media item or items in multiple ways,including different types of applications. For example, the mediaapplication 103 may provide the user with an option to create a link tosend to another person, generate a shared album that may be shared withanother person, add the media item or items to a preexisting sharedalbum, or share the media item or items with one or more messagingapplications, messing applications, chat applications, emailapplications, cloud-computing storage, map applications, text-editingapplications, etc. In some implementations, the media application 103may suggest the mode of sharing based on the user's frequency of user,another person's frequency of use, etc. In some implementations wherethe user selects a messaging application, the media application 103 mayprovide a list of users and an indication of availability for the users.

In some implementations, the media application 103 may provide an optionfor the user to add text to the media item or items, such as a caption.In some implementations, the media application 103 may provide an optionfor the user to associate emojis with the media item or items. In someimplementations, when the media item or items are delivered to a person,the media application 103 may display one emoji morphing into anotheremoji. For example, when a first user provides the caption “And it wasgreat!!!

Except, none of that happened

,” the media application 103 may display the caption as “And it wasgreat!!!” with the smiley faced emoji morphing into the sad-faced emojiand then the caption “Except, none of that happened.”

In some implementations, the media application 103 may provide an optionto share the media with people that are nearby and/or people indifferent regional locations, such as a town, country, etc. In someimplementations, the media application 103 may provide an option toshare the media offline. For example, the media application 103 may savethe media to backup storage and share the media via SMS.

In some implementations, the media application 103 may, upon userconsent, perform object identification and identification of people inthe images. The media application 103 may, upon user consent, provide anoption for the user to search the user's media based on objects, people,places, things, etc. In some implementations, the media application 103may provide the user with an option to search for different types ofthings in the images by providing the user with example images.

In some implementations, the media application 103 may include an imageassistant that detects problems with images captured by a user. Forexample, the images may be overexposed, blurry, etc. The mediaapplication may provide the user with an option to edit the image, forexample, as included in FIG. 4. In some implementations, the mediaapplication 103 provides suggestions to the user, such as a warning thatthe images are blurry or the user is taking images that are similar toimages the user has already captured. In some implementations, the mediaapplication 103 automatically corrects the images for the user.

In some implementations, the media application 103 provides an option togenerate a photobook from media associated with a user.

Example Method

FIG. 7 illustrates a flowchart of an example method 700 to generate acomposite image from a physical item according to some implementations.The method 700 is performed by a media application 103 stored on acomputing device 300, such as a mobile device 115, a media server 111,or in part a mobile device 115 and in part a media server 111.

At block 702, a first image of a physical item is captured with a camera243. The camera 243 is part of the computing device 300. The physicalitem is, for example, a photograph.

At block 704, borders associated with the physical item are recognized.For example, the computing device 300 identifies the four edges of thephotograph.

At block 706, based on the first image, an overlay is generated thatincludes a plurality of objects that are positioned within one or moreof the borders associated with the physical item. For example, the mediaapplication 103 divides the photograph into quadrants and generates anoverlay with four objects that are each centered within one of thequadrants.

At block 708, subsequent images of the physical item are captured withthe camera 243, where each subsequent image is captured with arespective subsequent camera position. For example, the mediaapplication 103 determines the movement of the camera 243 each time asubsequent image is captured. At block 710, during capture of thesubsequent images, an image preview is displayed that includes theoverlay, where the plurality of objects remains fixed in the overlay asthe camera 243 is moved, in response to a predetermined number ofcaptures of subsequent images at each respective subsequent cameraposition, an attribute of an object of the plurality of objects isupdated, and updating the attribute of the plurality of objectscorresponds to a completion of capture. For example, as a user moves thecamera 243 a particular way, a respective object of the four objectschanges color until the subsequent objects have been captured. In someimplementations, the overlay includes a reticle that moves when thecamera moves 243 and the reticle is moved to surround or overlap with atleast a portion of each of the plurality of objects until the subsequentimages have been captured and the result is a completion of capture.

At block 712, correspondence is established between pixels of the firstimage and pixels of the subsequent images. For example, the mediaapplication 103 compares all pixels at a particular location in thefirst image and the second image. At block 714, a composite image of thephysical item is generated where each pixel value of the composite imageis based on corresponding pixel values of the first image and thesubsequent images. For example, the media application 103 selects theminimum value of the corresponding pixel values for the composite imagefor that particular location.

FIG. 8 illustrates a flowchart of another example method 800 to generatea composite image. The method 800 is performed by a media application103 stored on a computing device 300, such as a mobile device 115, amedia server 111, or in part a mobile device 115 and in part a mediaserver 111.

At block 802, a first image of a physical item is captured with acamera. At block 804, the first image is displayed with a color overlayover a first portion of the image. At block 806 one or more subsequentimages of the physical item are captured, where each subsequent image iscaptured as the camera is moved with reference to the physical item. Atblock 808, during capture of each of the subsequent images, the firstimage is displayed with the color overlay over a second portion of thefirst image, where the second portion includes the first portion and anadditional portion determined based on movement of the camera. At block810, correspondence is established between pixels of the first image andpixels of each of the one or more subsequent images. At block 812 acomposite image of the physical item is generated, where each pixelvalue of the composite image is selected based on corresponding pixelvalues of the first image and the one or more subsequent images.

In the above description, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofthe specification. It will be apparent, however, to one skilled in theart that the disclosure can be practiced without these specific details.In some instances, structures and devices are shown in block diagramform in order to avoid obscuring the description. For example, theimplementations can be described above primarily with reference to userinterfaces and particular hardware. However, the implementations canapply to any type of computing device that can receive data andcommands, and any peripheral devices providing services.

Reference in the specification to “some implementations” or “someinstances” means that a particular feature, structure, or characteristicdescribed in connection with the implementations or instances can beincluded in at least one implementation of the description. Theappearances of the phrase “in some implementations” in various places inthe specification are not necessarily all referring to the sameimplementations.

Some portions of the detailed descriptions above are presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic data capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these data as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms including “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission, or display devices.

The implementations of the specification can also relate to a processorfor performing one or more steps of the methods described above. Theprocessor may be a special-purpose processor selectively activated orreconfigured by a computer program stored in the computer. Such acomputer program may be stored in a non-transitory computer-readablestorage medium, including, but not limited to, any type of diskincluding optical disks, ROMs, CD-ROMs, magnetic disks, RAMs, EPROMs,EEPROMs, magnetic or optical cards, flash memories including USB keyswith non-volatile memory, or any type of media suitable for storingelectronic instructions, each coupled to a computer system bus.

The specification can take the form of some entirely hardwareimplementations, some entirely software implementations or someimplementations containing both hardware and software elements. In someimplementations, the specification is implemented in software, whichincludes, but is not limited to, firmware, resident software, microcode,etc.

Furthermore, the description can take the form of a computer programproduct accessible from a computer-usable or computer-readable mediumproviding program code for use by or in connection with a computer orany instruction execution system. For the purposes of this description,a computer-usable or computer-readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

A data processing system suitable for storing or executing program codewill include at least one processor coupled directly or indirectly tomemory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

In situations in which the systems discussed above collect or usepersonal information, the systems provide users with an opportunity tocontrol whether programs or features collect user information (e.g.,information about a user's social network, social actions or activities,profession, a user's preferences, or a user's current location), orcontrol whether and/or how to receive content from the server that maybe more relevant to the user. In addition, certain data may be treatedin one or more ways before it is stored or used, so that personallyidentifiable information is removed. For example, a user's identity maybe treated so that no personally identifiable information can bedetermined for the user, or a user's geographic location may begeneralized where location information is obtained (such as to a city,ZIP code, or state level), so that a particular location of a usercannot be determined. Thus, the user may have control over howinformation is collected about the user and used by the server.

What is claimed is:
 1. A method comprising: capturing, with a camera, afirst image of a physical item at a first camera position; detectingedges associated with the physical item; based on the first image,generating an overlay that includes a first object that is positionedwithin one or more of the edges associated with the physical item;capturing, with the camera, subsequent images of the physical item,wherein each subsequent image is captured with a respective subsequentcamera position; during capture of the subsequent images, displaying animage preview that includes the overlay; establishing correspondencebetween pixels of the first image and pixels of each of the subsequentimages; and generating a composite image of the physical item, whereineach pixel value of the composite image is based on corresponding pixelvalues of the first image and the subsequent images and the compositeimage depicts reduced or eliminated defects that occur when capturingone or more of the first image and the subsequent images.
 2. The methodof claim 1, wherein during capture of the subsequent images: the firstobject remains fixed in the overlay as the camera is moved; and inresponse to a predetermined number of captures of subsequent images ateach respective subsequent camera position, removing the first objectfrom the overlay to indicate a completion of capture of the subsequentimages associated with the respective subsequent camera position.
 3. Themethod of claim 1, wherein: the overlay further includes a secondobject; the overlay further comprises a reticle that moves from thefirst object to the second object as the camera is moved; and thesubsequent images are captured responsive to at least a portion of thereticle overlapping with the first object and then overlapping with thesecond object.
 4. The method of claim 3, wherein the overlay furtherincludes an arrow that directs the camera to be moved from the firstobject to the second object.
 5. The method of claim 3, wherein thereticle is illustrated as a spinner that progressively changes toindicate an amount of time needed for the camera to capture a subset ofthe subsequent images while at least the portion of the reticle overlapswith the second object.
 6. The method of claim 3, wherein: the reticleis displayed at a fixed size and the first object is displayed at anobject size that corresponds to a distance between the physical item andthe camera; the object size of the first object increases in the overlayas the camera is moved closer to physical item; and the object sizecorresponds to the fixed size of the reticle when the camera is at anoptimal distance from the physical item.
 7. The method of claim 1,wherein responsive to the first camera position for the first imagebeing an angle, the overlay is displayed with the first object at theangle.
 8. A non-transitory computer-readable medium with instructionsthat, when executed by one or more computers, cause the one or morecomputers to perform operations comprising: capturing, with a camera, afirst image of a physical item at a first camera position; detectingedges associated with the physical item; based on the first image,generating an overlay that includes a first object that is positionedwithin one or more of the edges associated with the physical item;capturing, with the camera, subsequent images of the physical item,wherein each subsequent image is captured with a respective subsequentcamera position; during capture of the subsequent images, displaying animage preview that includes the overlay; establishing correspondencebetween pixels of the first image and pixels of each of the subsequentimages; and generating a composite image of the physical item, whereineach pixel value of the composite image is based on corresponding pixelvalues of the first image and the subsequent images and the compositeimage depicts reduced or eliminated defects that occur when capturingone or more of the first image and the subsequent images.
 9. Thecomputer-readable medium of claim 8, during capture of the subsequentimages: the first object remains fixed in the overlay as the camera ismoved; and in response to a predetermined number of captures ofsubsequent images at each respective subsequent camera position,removing the first object from the overlay to indicate a completion ofcapture of the subsequent images associated with the respectivesubsequent camera position.
 10. The computer-readable medium of claim 8,wherein: the overlay further includes a second object; the overlayfurther comprises a reticle that moves from the first object to thesecond object as the camera is moved; and the subsequent images arecaptured responsive to at least a portion of the reticle overlappingwith the first object and then overlapping with the second object. 11.The computer-readable medium of claim 10, wherein the overlay furtherincludes an arrow that directs the camera to be moved from the firstobject to the second object.
 12. The computer-readable medium of claim10, wherein the reticle is illustrated as a spinner that progressivelychanges to indicate an amount of time needed for the camera to capture asubset of the subsequent images while at least the portion of thereticle overlaps with the second object.
 13. The computer-readablemedium of claim 10, wherein: the reticle is displayed at a fixed sizeand the first object is displayed at an object size that corresponds toa distance between the physical item and the camera; the object size ofthe first object increases in the overlay as the camera is moved closerto physical item; and the object size corresponds to the fixed size ofthe reticle when the camera is at an optimal distance from the physicalitem.
 14. The computer-readable medium of claim 8, wherein responsive tothe first camera position for the first image being an angle, theoverlay is displayed with the first object at the angle.
 15. A systemcomprising: one or more processors; and a memory that storesinstructions executable by the one or more processors that, whenexecuted by the one or more processors, cause the one or more processorsto perform operations comprising: capturing, with a camera, a firstimage of a physical item at a first camera position; detecting edgesassociated with the physical item; based on the first image, generatingan overlay that includes a first object that is positioned within one ormore of the edges associated with the physical item; capturing, with thecamera, subsequent images of the physical item, wherein each subsequentimage is captured with a respective subsequent camera position; duringcapture of the subsequent images, displaying an image preview thatincludes the overlay; establishing correspondence between pixels of thefirst image and pixels of each of the subsequent images; and generatinga composite image of the physical item, wherein each pixel value of thecomposite image is based on corresponding pixel values of the firstimage and the subsequent images and the composite image depicts reducedor eliminated defects that occur when capturing one or more of the firstimage and the subsequent images.
 16. The system of claim 15, whereinduring capture of the subsequent images: the first object remains fixedin the overlay as the camera is moved; and in response to apredetermined number of captures of subsequent images at each respectivesubsequent camera position, removing the first object from the overlayto indicate a completion of capture of the subsequent images associatedwith the respective subsequent camera position.
 17. The system of claim15, wherein: the overlay further includes a second object; the overlayfurther comprises a reticle that moves from the first object to thesecond object as the camera is moved; and the subsequent images arecaptured responsive to at least a portion of the reticle overlappingwith the first object and then overlapping with the second object. 18.The system of claim 17, wherein the overlay further includes an arrowthat directs the camera to be moved from the first object to the secondobject.
 19. The system of claim 17, wherein the reticle is illustratedas a spinner that progressively changes to indicate an amount of timeneeded for the camera to capture a subset of the subsequent images whileat least the portion of the reticle overlaps with the second object. 20.The system of claim 17, wherein: the reticle is displayed at a fixedsize and the first object is displayed at an object size thatcorresponds to a distance between the physical item and the camera; theobject size of the first object increases in the overlay as the camerais moved closer to physical item; and the object size corresponds to thefixed size of the reticle when the camera is at an optimal distance fromthe physical item.